Adaptive screen and video coding scheme

ABSTRACT

An adaptive screen and video coding system adaptively selects different coding schemes for coding an image block of a screen or video image based on a type of content included in the image block. The adaptive screen and video coding system further includes a screen coding scheme that classifies pixels of the image block into major colors and escape pixels and convert values of the pixels into index values assigned to the major colors and the escape pixels. Since the number of index values is usually less than a normal range of pixel values, using the index values instead of actual pixel values further facilitate compression and coding of the image block, thus improving coding accuracy and efficiency of the screen or video image. In one embodiment, the adaptive screen and video coding system may be used for screen sharing application.

BACKGROUND

With the rapid development of computer technologies, screen sharingapplications, which were once limited to enterprise applications such asremote desktop control of office computers, have now become available tocommon users for entertainment and communications, for example. Thesescreen sharing applications usually involve at least two computingdevices which necessarily communicate screen data over a data networkbetween the two computing devices. In order to provide real-timeinteractive screen sharing experience and reduce the amount of data sentover the network, the screen data is usually encoded and compressedprior to sending. Conventional hybrid video coding schemes, such asH.264 and HEVC (High Efficiency Video Coding), which are based ontransform domain quantization and entropy coding, have been widely usedfor this purpose. These hybrid video coding schemes, however, arenormally efficient for encoding nature videos, but fail to provideefficient coding and compression results for other types of content suchas text content or other high-gradient content which normally constitutea large portion of the screen data. Therefore, unless a coding schemethat can adaptively and efficiently perform coding and compression for avariety of types of content is developed, these screen sharingapplications, though implementable, may fail to achieve a screen sharingperformance that is satisfactory to most users.

SUMMARY

This summary introduces simplified concepts of adaptive screen and videocoding, which are further described below in the Detailed Description.This summary is not intended to identify essential features of theclaimed subject matter, nor is it intended for use in limiting the scopeof the claimed subject matter.

This application describes example embodiments of adaptive screen andvideo coding. In one embodiment, a system receives an image, e.g., animage frame of a screen or a video, and partitions the image into aplurality of input blocks. The system examines the input blocks anddetermines which coding is used for each input block. Examples of codingalgorithms include, for example, algorithm a screen coding algorithm(which is specific for encoding and compressing text content and otherhigh-gradient content) or another video coding algorithm. In oneembodiment, the system may determine which coding algorithm is used foreach input block based on, for example, respective prediction accuraciesand/or coding efficiencies that the screen coding algorithm and thevideo coding algorithm may achieve for each input block. In response todetermining which coding algorithm is used for each input block, thesystem may adaptively apply respective coding algorithms for each inputblock to complete coding and compression for the image.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is set forth with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical items.

FIG. 1 illustrates an example adaptive screen and video codingframework.

FIG. 2 illustrates an example environment of an adaptive screen andvideo coding system.

FIG. 3 illustrates the example adaptive screen and video coding systemas shown in FIG. 2.

FIG. 4 illustrates an example graphical representation of colorclassification.

FIG. 5 illustrates an example line-based prediction method.

FIG. 6 illustrates an example method of adaptive screen and videocoding.

FIG. 7 illustrates an example method of predictive coding.

DETAILED DESCRIPTION Overview

As noted above, existing encoding and compression algorithms fail toadaptively encode and/or compress different types of content (e.g.,text, graphics, pictures, videos, etc.) within a single image and/oracross multiple images adaptively, and hence fail to provide anefficient coding and compression result for screen sharing applications.

This disclosure describes an adaptive screen and video coding system.The adaptive screen and video coding system receives an image (such asan image frame of a screen or a video) and adaptively selects a codingand compression algorithm for the image or an image portion of the imagebased on a type of content within the image or the image portion. Theselection may be based on, for example, coding/compression performanceof the coding and compression algorithm for that image or the imageportion. The coding/compression performance may include predictionaccuracy, coding/compression efficiency associated with the image or theimage portion, for example.

FIG. 1 shows an example adaptive screen and video coding framework 100that may be used by the adaptive screen and video coding system.Although this example framework 100 is constructed based on a HEVC videocoding scheme, the adaptive screen and video coding system is notlimited to HEVC, but may also be applicable to other image and/or videocoding schemes or standards such as JPEG, JPEG2000, H.261, H.263, H.264,MPEG-1. MPEG-2, MPEG-4, etc. In this example, the framework 100 mayinclude modules such as a coding module 102, a transformation module104, a quantization module 106, an entropy coding module 108, an inversetransformation module 110, an inverse quantization module 112, adeblocking filter 114, a sample adaptive offset 116 and a decodedpicture buffer 118. Details of the transformation module 104, thequantization module 106, the entropy coding module 108, the inversetransformation module 110, the inverse quantization module 112, thedeblocking filter 114, the sample adaptive offset (SAO) filter 116 andthe decoded picture buffer 118 will be described further below.

In one embodiment, the adaptive screen and video coding system mayreceive an image. The received image may include an image frame of ascreen or a video. Upon receiving the image, the adaptive screen andvideo coding system may partition or divide the image into a pluralityof input blocks. In one embodiment, the adaptive screen and video codingsystem may determine which coding scheme (e.g., a screen coding scheme120, an intra-picture prediction scheme 122, an inter-picture predictionscheme 124, etc.) is to be applied for each input block. In someembodiments, the adaptive screen and video coding system may determinewhich coding scheme to be applied for each input block based onrespective prediction accuracies and/or coding efficiencies of thecoding schemes. Additionally or alternatively, the adaptive screen andvideo coding system may determine which coding scheme is to be appliedfor each input block based on which coding scheme has been applied for arespective input block that is located at a same position or within aneighborhood thereof in a previous image (e.g., a previous image frameof the screen or the video, etc.).

In one instance, responsive to determining that the screen coding scheme120 is used for a particular input block, the adaptive screen and videocoding system may perform color classification. In one embodiment, theadaptive screen and video coding system may compute a distribution ofpixel values (or intensities), e.g., a histogram, of the input block.After obtaining the distribution of pixel values for the input block,the adaptive screen and video coding system may select one or more pixelvalues from the distribution as one or more major colors. In oneembodiment, the one or more major colors are one or more representativecolor or pixel values for (all or a majority of) pixels included in theinput block. By way of example and not limitation, the one or more majorcolors may include, for example, a pixel value having frequencycorresponding to one of first N highest frequencies within thedistribution of pixel values (where N is a positive integer), a pixelvalue having frequency greater than or equal to a predeterminedfrequency threshold, and/or pixel values of having a total frequencygreater than or equal to a predetermined total frequency threshold, etc.In some embodiments, the adaptive screen and video coding system maytreat pixel values other than one or more major colors as escape pixels.The escape pixels correspond to pixel values that appear in the inputblock less frequently, for example, as compared to the one or more majorcolors.

In one embodiment, the adaptive screen and video coding system mayassign a unique index value to each major color. Additionally oralternatively, the adaptive screen and video coding system may assignone or more unique index values to the escape pixels. Upon selecting theone or more major colors for the input block, the adaptive screen andvideo coding system classifies or categorizes each pixel within theinput block into a major color or an escape pixel based on its pixelvalue, and converts each pixel of the input block into a respectiveindex value based on which major color or escape pixel the pixel isclassified as.

In some embodiments, after converting pixel values of the pixels of theinput block into respective index values (or an index value map), theadaptive screen and video coding system may perform prediction andcoding. By way of example and not limitation, the adaptive screen andvideo coding system may apply a compression algorithm to compress andcode the index value map of that particular input block. In oneembodiment, the compression algorithm may include a plurality of modesand context models that facilitate prediction of a current pixel orpixel line based on a previous pixel or pixel line, and hence representsthe current pixel or pixel line using a short code based on the previouspixel or pixel line.

Upon coding the input block, the adaptive screen and video coding systemmay perform operations such as quantization, transformation and entropycoding for encoding operations using the other modules of the adaptivescreen and video coding framework as described in the foregoingembodiments, and sending an encoding result to a client device fordecoding and/or display. The encoding result may include, for example,information of index values representing the major colors and escapepixels, codes obtained for each input block, etc., to a client devicefor decoding and/or display.

The described system adaptively or selectively determines a codingalgorithm for an input block of an image, and applies the determinedcoding algorithm to classify, predict and code pixels of the inputblock, thus improving accuracy and efficiency of coding different typesof content (such as text, graphic, etc.) included in the image.

In the examples described herein, the adaptive screen and video codingsystem determines a coding algorithm for an input block, classifiespixels within the input block into major colors and escape pixels,converts pixels values into index values, and predicts and codes theindex values. However, in other embodiments, these functions may beperformed by one or more services located at a same location ordifferent locations. For example, in at least one embodiment, adetermination service may determine a coding algorithm for an inputblock, while a classification service may classify pixels within theinput block into major colors and escape pixels. A conversion servicemay convert pixels values into index values, and a coding service maypredict and code the index values.

Furthermore, although in the examples described herein, the adaptivescreen and video coding system may be implemented as software and/orhardware installed in a single device or as a service, in otherembodiments, the adaptive screen and video coding system may beimplemented in a plurality of devices and/or services provided in one ormore servers over a network and/or distributed in a cloud computingarchitecture.

The application describes multiple and varied implementations andembodiments. The following section describes an example framework thatis suitable for practicing various implementations. Next, theapplication describes example systems, devices, and processes forimplementing an adaptive screen and video coding system.

Example Environment

FIG. 2 illustrates an example environment 200 usable to implement anadaptive screen and video coding system 202. In this example, theadaptive screen and video coding system 202 is described as included ina client device 204. However, in other instances, the adaptive screenand video coding system 202 may be an entity independent or separatefrom the client device 204. In some embodiments, the adaptive screen andvideo coding system 202 may be a peripheral device of the client device204. In some instances, the adaptive screen and video coding system 202may be included in and/or distributed among one or more server(s) 206,which communicate data with the client device 204 via a network 208.Additionally or alternatively, in some embodiments, the functions of theadaptive screen and video coding system 202 may be included in and/ordistributed among the client device 204 and one or more servers 206. Forexample, the one or more servers 206 may include part of the functionsof the adaptive screen and video coding system 202 while other functionsof the adaptive screen and video coding system 202 may be included inthe client device 204. Furthermore, in some embodiments, some or all thefunctions of the adaptive screen and video coding system 202 may beincluded in a cloud computing system or architecture.

The client device 204 may be implemented as any of a variety ofelectronic and/or computing devices that include, are associated with,or are connected to a display or screen. By way of example and notlimitation, the client device 204 may be implemented as any of a varietyof computing devices including, but not limited to, a mainframecomputer, a notebook or portable computer, a handheld device, a netbook,an Internet appliance, a tablet or slate computer, a mobile device(e.g., a mobile phone, a personal digital assistant, a smart phone,etc.), a game console, a set-top box, a video player, etc., or acombination thereof.

The network 208 may be a wireless or a wired network, or a combinationthereof. The network 208 may be a collection of individual networksinterconnected with each other and functioning as a single large network(e.g., the Internet or an intranet). Examples of such individualnetworks include, but are not limited to, telephone networks, cablenetworks, Local Area Networks (LANs), Wide Area Networks (WANs), andMetropolitan Area Networks (MANs). Further, the individual networks maybe wireless or wired networks, or a combination thereof. Wired networksmay include an electrical carrier connection (such a communicationcable, etc.) and/or an optical carrier or connection (such as an opticalfiber connection, etc.). Wireless networks may include, for example, aWiFi network, other radio frequency networks (e.g., Bluetooth®, Zigbee,etc.), etc.

In one embodiment, the client device 204 may include one or moreprocessing units 210 coupled to memory 212. The one or more processingunit(s) 210 may be implemented as one or more hardware processorsincluding, for example, a microprocessor, an application-specificinstruction-set processor, a graphics processing unit, a physicsprocessing unit (PPU), a central processing unit (CPU), a graphicsprocessing unit (GPU), a digital signal processor, etc. Additionally oralternatively, the functionality described herein can be performed, atleast in part, by one or more hardware logic components. For example,and without limitation, illustrative types of hardware logic componentsthat can be used include field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), application-specificstandard products (ASSPs), system-on-a-chip systems (SOCs), complexprogrammable logic devices (CPLDs), etc.

The memory 212 may include or store one or more applications 214 (e.g.,a screen sharing application, a video player application, etc.) that areexecuted by the one or more processing unit(s) 210, and other programdata 216. The memory 212 may be coupled to, associated with, and/oraccessible to other devices, such as network servers, routers, and/orthe other servers 206.

The memory 212 may include volatile memory, such as Random Access Memory(RAM) and/or non-volatile memory, such as read only memory (ROM) orflash RAM. The memory 212 is an example of computer-readable media.Computer-readable media includes at least two types of computer-readablemedia, namely computer storage media and communications media.

Computer storage media includes volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer readable instructions, data structures,program modules, or other data. Computer storage media includes, but isnot limited to, phase change memory (PRAM), static random-access memory(SRAM), dynamic random-access memory (DRAM), other types ofrandom-access memory (RAM), read-only memory (ROM), electricallyerasable programmable read-only memory (EEPROM), flash memory or othermemory technology, compact disk read-only memory (CD-ROM), digitalversatile disks (DVD) or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other non-transmission medium that can be used to storeinformation for access by a computing device.

In contrast, communication media may embody computer-readableinstructions, data structures, program modules, or other data in amodulated data signal, such as a carrier wave, or other transmissionmechanism. As defined herein, computer storage media does not includecommunication media.

A user 218 may use the application 214 (such as the screen sharingapplication, etc.) of the client device 204 to remotely connect toanother computing device via the network 208. The user 218 may want totransmit or share screen data that is presented in a display 220 of theclient device 204 to the other computing device via the network 208. Thescreen data may include a variety of content including, for example,text content, graphic content, video content, image content, etc. Theapplication 214 may perform this transmission or sharing of the screendata in an accurate and efficient manner using the adaptive screen andvideo coding system 202.

Example Adaptive Screen and Video Coding System

FIG. 3 illustrates an example adaptive screen and video coding system202 in more detail. In this example, the example adaptive screen andvideo coding system 202 is described to be included in or part of aclient device 204. As described in the foregoing, the client device 204may include, but is not limited to, one or more processing units 210 andmemory 212. Additionally, the client device 204 may further include oneor more applications 214. In some embodiments, the client device 204 mayfurther include a network interface 302 and an input/output interface204. The one or more processing units 210 are configured to executeinstructions received from the network interface 302, received from theinput/output interface 304, and/or stored in the memory 212. In oneembodiment, the client device 204 further includes a display 220. Thedisplay 220 may include a touch screen, a normal screen (i.e., a screenwithout touch-sensitive capability), etc.

The adaptive screen and video coding system 202 may include programmodules 306 and program data 308. In one embodiment, the adaptive screenand video coding system 202 may include an input module 310 and a codingmodule 102. In some embodiments, the adaptive screen and video codingsystem 202 may additionally include a transformation module 104, aquantization module 106 and an entropy coding module 108 for encodingoperations. Additionally, in one embodiment, the adaptive screen andvideo coding system 202 may further include an inverse transformationmodule 110, an inverse quantization module 112, a deblocking filter 114,a sample adaptive offset (SAO) filter 116 and a decoded picture buffer118 for decoding operations. Additionally, the adaptive screen and videocoding system 202 may further include an output module 312 and otherprogram data 314.

In one embodiment, the input module 310 is configured to receive inputdata. The input module 310 may receive this input data from theapplication 214 (e.g., a screen sharing application) and/or the display220 (e.g., a touch screen display), etc. The image data may include oneor more types of content including, but not limited to, text, graphics,a picture, an image frame of a video, etc. In response to receiving theinput data, the input module 310 may partition or divide the input datainto a plurality of input blocks (or coding units in HEVC, for example).

Upon partitioning or dividing the input data, the coding module 102 ofthe adaptive screen and video coding system 202 may compress and/or codethe plurality of input blocks. In one embodiment, the coding module 102may include a plurality of coding schemes for different types ofcontent. By way of example and not limitation, the plurality of codingschemes may include a screen coding scheme, an intra-picture predictioncoding scheme, an inter-picture prediction coding scheme, etc. In oneembodiment, the coding module 102 may determine which coding scheme isused for an input block based on, for example, respective accuraciesand/or coding efficiencies of the plurality of coding schemes for thatinput block. For example, the coding module 102 may code the input blockinto a plurality of codes using the plurality of coding schemes andselect a coding scheme that produces a code having the highest accuracyand/or the highest compression rate among the plurality of codingschemes.

Additionally or alternatively, the coding module 102 may determine orselect which coding scheme is to be used for an input block based on atype of content that is included in the input block. By way of exampleand not limitation, the coding module 102 may select the screen codingscheme in response to determining that at least a predetermined amountor percentage (e.g., 60%, 70%, 80%, 95%, etc.) of content in the inputblock corresponds to text content and/or other high-gradient content,and select a video coding scheme such as the intra-picture predictioncoding scheme. Otherwise, the coding module 102 may select theinter-picture prediction coding scheme.

Additionally or alternatively, the coding module 102 may determine orselect a coding scheme for coding a current input block based on whichcoding scheme(s) has/have been selected for coding one or more otherinput blocks that are adjacent and/or previous to the current inputblock within a same frame and/or a previous frame of the input data. Forexample, the coding module 102 may select a coding scheme for a currentinput block if the coding scheme is used for a majority of input blocksthat are within a neighborhood (e.g., currently adjacent to the currentinput block, etc.) of the current input block within a same frame and/ora previous frame of the input data. In some embodiments, the codingmodule 102 may perform one or more other strategies (such as strategiesthat are based on accuracy, efficiency, type of content, etc.) asdescribed above if the majority of input blocks fails to reach a numberor percentage (e.g., 30% or above) predetermined by the user 218 and/orthe adaptive screen and video coding system 202.

Upon determining or selecting a coding scheme for coding an input block,the coding module 102 may perform an associated coding based on thedetermined or selected coding scheme. In some embodiments, the codingmodule 102 may determine or select a video coding scheme such as theintra-picture prediction coding scheme or the inter-picture predictioncoding scheme, and perform coding based on intra-picture prediction orinter-picture prediction.

In one embodiment, if the screen coding scheme is selected for a currentinput block, the coding module 102 may perform color classification (orpixel classification), and prediction/coding according to the screencoding scheme. In one embodiment, the coding module 102 may compute adistribution of pixel values (or intensities) within the current inputblock. For example, the coding module 102 may compute or countfrequencies of appearance or existence of different pixel values withinthe current input block.

In response to obtaining the distribution of pixel values within thecurrent input block, the coding module 102 may classify pixels of thecurrent input block into different pixel classes. By way of example andnot limitation, the coding module 102 may classify the pixels into oneor more major colors (or major pixel colors or values) and one or moreescape pixels. In one embodiment, the coding module 102 may classify thepixels with respect to each color component. For example, a color valueor intensity of a pixel may be represented in three different colorcomponents (such as green, red, blue, etc.), and the coding module 102may classify each color component of the pixel into a major color or anescape pixel for that color component. In this first scenario, thecoding module 102 may determine one or more major colors and one or moreescape pixels for each color component. In some embodiments, the codingmodule 102 may classify the pixels based on respective overall color orpixel values in a color model (such as a RGB (Red, Green and Blue) colormodel, a CMYK (Cyan, Magenta and Yellow) color model, a YUV (i.e., oneluma component Y, and two chrominance components U and V) color model,etc.). In this second scenario, the coding module 102 may determine oneor more major colors and one or more escape pixels for overall color orpixel values instead of individual color components. The description orsection hereinafter applies to one or both of these scenarios for majorcolors and escape pixels.

In one embodiment, the one or more major colors may include one or moremost representative pixel values that cover a majority of pixels withinthe current input block. Additionally or alternatively, the one or moremajor colors may include, for example, a pixel value having frequencycorresponding to one of the N highest frequencies within thedistribution of pixel values (where N is a positive integer), a pixelvalue having frequency greater than or equal to a predeterminedfrequency threshold, and/or pixel values having a total frequencygreater than or equal to a predetermined total frequency threshold, etc.In some embodiments, the one or more escape pixels correspond to pixelvalues that are not included in the one or more major colors.

Additionally or alternatively, the coding module 102 may partition ordivide the pixel values of the current input block into one or moreclusters or sections based on a clustering algorithm (such as K-meansclustering, density-based clustering, etc.) or based on an even divisionor partitioning of pixel value range. In one embodiment, the codingmodule 102 may select a pixel value having the highest frequency ofappearance in each cluster or section as the one or more major colors.In some instances, the coding module 102 may select a pixel value havingthe highest frequency of appearance in each cluster or section as theone or more major colors (only) if the frequency of that pixel value isgreater than or equal to a predetermined frequency threshold, e.g.,five, ten, fifteen, etc.

In some embodiments, the coding module 102 may create a quantizationzone for a major color. The quantization zone may include or correspondto a range of pixel values around or in a neighborhood of the majorcolor, e.g., a range including m±q, where m corresponds to a pixel valueof the major color and q corresponds to a quantization value, e.g., one,two, five, ten, etc. For example, for lossy coding, the coding module102 may quantize pixel values that are within a predeterminedneighborhood of a major color to the value of the major color. Thecoding module 102 may set pixel values that do not belong to the one ormore major colors as escape pixels, which may also be quantized prior tocoding. In some examples, such as lossless coding, the coding module 102may not employ quantization.

In some embodiments, the coding module 102 may further allow a majorcolor to cover or represent a single pixel value or a range of pixelvalues. In one embodiment, the coding module 102 may allow one majorcolor to cover or represent a single pixel value and another major colorto cover or represent a range of pixel values. By way of example and notlimitation, the coding module 102 may allow a selected major color tocover or represent pixel values of a same cluster or section thereof. Insome instances, the coding module 102 may determine whether a majorcolor covers or represents a single pixel value or a range of pixelvalues based on a pixel density about the major color (e.g., a number ofpixels having pixel values within a predetermined neighborhood of themajor color) as determined from the distribution of pixel valuesobtained in the foregoing embodiments. For example, if the pixel densityabout a major color is greater than or equal to a predetermined densitythreshold (e.g., two, five, ten, etc.), the coding module 102 may employthe major color to cover or represent pixel values within apredetermined neighborhood (±2 pixel values in a pixel value range of256, for example) of the major color. Otherwise, the coding module 102may employ the major color to cover or represent a single pixel value towhich the major color corresponds. Alternatively, in some embodiments,the coding module 102 may perform quantization for pixel values andemploy each major color to cover or represent a range of pixel valueswithin a respective predetermined quantization zone.

In this example, the adaptive screen and video coding system 202 isdescribed to determine or select one or more major colors for each inputblock, and hence the one or more major colors may be different or variedfrom one input block to another input block within a same image or inputdata. In some embodiments, the adaptive screen and video coding system202 may determine one or more major colors for an entire image or inputdata, and employ these major colors for the entire image or input data.This may further save or reduce the amount of data transmitted bysending information of the major colors once for all input blocks of theimage or input data instead of sending information of respective majorcolors for different input blocks of the image or input data.

FIG. 4 illustrates an example classification result 400 obtained bycolor classification of the screen coding scheme. The classificationresult 400 in this example is represented as a histogram of pixel valueswhich ranges from zero to two hundred and fifty-five. Depending on acolor model or system, such as a RGB (Red, Green and Blue) color model,a CMYK (Cyan, Magenta and Yellow) color model, a YUV (i.e., one lumacomponent Y, and two chrominance components U and V) color model, etc.,that may be used, the range of pixel values may be varied from one modelto another model. In this example, the coding module 102 selects a pixelvalue having the highest frequency within a cluster or a quantizationzone 402 thereof as a major color 404. The coding module 102 may thentreat pixel values that are outside the clusters or quantization zones402 as escape pixels 406.

In response to determining the one or more major colors, the codingmodule 102 may assign color index values to the one or more major colorsand the one or more escape pixels. Since not every pixel value within apixel value range appears in the current input block, representing themajor colors and escape pixels as color index values reduces the amountof data used to code information of the current input block. In oneembodiment, the coding module 102 may include or select a same set ofindex for each color component of a color model (such as a RGB colormodel, a CMYK (color model, a YUV color model, etc.) for consistencyand/or easy of indexing. In some embodiments, the coding module 102 mayinclude or select a different set of index for each color component of acolor model. For example, the distribution of pixel values for eachcolor component of the color model may be different. The coding module102 may therefore select or determine a different set of index for eachcolor component of the color model to

In one embodiment, the coding module 102 may represent a color indexvalue of a pixel as a combination of a color index value of each colorcomponent of the pixel. For example, the coding module 102 may representa color index value as a single joint or concatenated index value ofcolor components, e.g., RGB. By way of example and not limitation, thecoding module 102 may generate a joint or concatenated index value usingthe following formula:

index_(R)<<2*bits_(index)+index_(G)<<2*bits_(index)+index_(B)=index_(joint)  (1)

where index_(R), index_(G) and index_(B) are color index values of red,green and blue for a same pixel, and index_(joint) corresponds to ajoint or concatenated index value for that same pixel.

In some embodiments, for a YUV color model or format, the coding module102 may perform joint index value coding based on 4:4:4 sampling, withjoint index value coding used only for U and V chrominance componentsfor 4:2:0 and 4:2:2 sampling. In one embodiment, the coding module 102may alternatively represent a color index value of each color componentof a pixel individually or independently without combining orconcatenating the color index value of each color component of the pixelinto a single index value (such as a joint index value as describedabove).

In one embodiment, the coding module 102 may classify the pixels of thecurrent input block into a major color or an escape pixel. Depending onwhich definition of a major color is used, the coding module 102 mayclassify the pixels of the current input block into a major color or anescape pixel based on respective pixel values accordingly. In oneembodiment, after classifying the pixels of the current input block, thecoding module 102 may convert the pixel values of the current inputblock into corresponding color index values (such as joint index values)that represent the major colors and the escape pixels respectively toform an index value map. In some instances, when the coding module 102represents a color index value of each color component of a pixelindividually or independently, the coding module 102 may convert pixelvalues in each color component to form a separate index value mapassociated with that color component for the pixels. Additionally, thecoding module 102 may include information of these color index valuesthat represent the major colors and the escape pixels in a final code ofthe current input block so that a receiver of the code may determinewhich pixel values these color index values represent.

In one embodiment, after classifying the pixels and converting the pixelvalues into the color index values to form an index value map of thecurrent input block (or an index value map of each color component), thecoding module 102 may exploit or determine correlations among the pixelsof the current input block for compressing and coding pixel informationof the current input block. After forming the index value map of thecurrent input block, the coding module 102 may employ a predictionalgorithm to exploit or determine the correlations among the pixels ofthe current input block. In one embodiment, the coding module 102 mayemploy a line-based prediction method or a block-based predictionmethod. The line-based prediction method or algorithm predicts a currentpixel line (i.e., a current sequence of pixel values along a horizontalline of an input block) based on another pixel line and/or one or morepixel values that are related or adjacent to the current pixel line.FIG. 5 illustrates a graphical representation 500 of the line-basedprediction method. In one embodiment, the line-based prediction methodmay include a plurality of prediction or line modes 502 used to signal aprediction approach for a current pixel line 504 of an index value map506 associated with a current input block. Examples of the plurality ofprediction or line modes 502 may include a horizontal mode, a verticalmode and a normal mode.

By way of example and not limitation, the horizontal mode may representthat each pixel along a same pixel line has a same pixel value. In oneembodiment, if this pixel value is the same as a first pixel of a pixelline that is previous to or above of the current pixel line, the codingmodule 102 may represent or code this current pixel line using data(such as mode bits) representing the horizontal mode without codinginformation about the pixel value. In some embodiments, if this pixelvalue is different from a first pixel of a pixel line that is previousto or above of the current pixel line, the coding module 102 mayrepresent or code this current pixel line using the data representingthe horizontal mode and additional data representing the pixel value ofthe current pixel line.

Additionally, the vertical mode may represent that a current pixel lineis the same as a pixel line that is immediately prior to or above thecurrent pixel line. In this instance, the coding module 102 mayrepresent or code this current pixel line using data representing thevertical mode without including information of pixel values of thecurrent pixel line as this information has been represented or codedpreviously, for example, for the above pixel line. In some embodiments,the normal mode may represent that pixels in a current pixel line arepredicted individually. For example, the coding module 102 may employone or more neighbors (e.g., immediate neighbors) of a current pixel ofthe current pixel line as a predictor and code a prediction directionfor the current pixel based on the one or more neighbors.

In one embodiment, the coding module 102 may employ a fixed length codeor a variable length code to represent a prediction mode for a pixelline. By way of example and not limitation, the coding module 102 mayemploy one data unit or bin (such as one data bit, one data byte, etc.,depending on a smallest data unit employed by the client device 204 orthe adaptive screen and video coding system 202, for example) toindicate whether the normal mode is used, and an additional one dataunit to indicate whether the horizontal mode or the vertical mode isused. Additionally, the coding module 102 may employ one or more dataunits, e.g., two data units, to indicate a prediction direction for apixel in the normal mode.

In some embodiments, in order to further improve coding and compressionperformance, the line-based prediction method may further include one ormore context models 508. The line-based prediction method uses thecontext models to represent relationship(s) between a current pixel andits one or more neighbors, e.g., correlation between a pixel value ofthe current pixel and pixel values of its neighbors. By way of exampleand not limitation, the line-based prediction method may include twocontext models for a first data unit of the prediction mode and threecontext models for a second data unit of the prediction mode. FIG. 5also shows an example code 510 for the index value map 506 based on thedescribed line-based prediction method.

Due to a high-pixel dependency of the line-based prediction method, insome embodiments, the coding module 102 may alternatively employ theblock-based prediction method for performing predictive coding of thecurrent input block. The algorithm for the block-based prediction methodis similar to that of the line-based prediction method, except that apixel sub-block (such as a 4×4 block, a 8×8 block, etc.) is used insteadof a pixel line within a current input block and no pixel predictiondependency is allowed or employed within the pixel sub-block.

In order to increase the flexibility and further improve the performanceof the line-based method, in some embodiments, a pixel line may vary inlength. For example, the coding module 102 may divide a current pixelline within the current input block into a plurality of pixel sub-lines,and separately or individually determine which line mode is to be usedfor a pixel sub-line of the current pixel line. In other words, thecoding module 102 may determine that more than one line mode, e.g., thehorizontal mode and the vertical mode, may be used for the current pixelline pixel line, a line mode for a pixel sub-line and another line modefor another pixel sub-line of the same pixel line, for example. Forexample, the coding module 102 may signal or select the vertical modefor a portion of a current pixel line (i.e., a pixel sub-line) thatincludes one or more consecutive pixels in the current pixel line if theone or more consecutive pixels are the same with their respectiveneighbors above. Additionally or alternatively, the coding module 102may signal or select the horizontal mode for a portion of a currentpixel line (i.e., a pixel sub-line) that includes one or moreconsecutive pixels in the current pixel line if the one or moreconsecutive pixels have the same index values (or pixel values). In oneembodiment, the coding module 102 may record and transmit associatednumber of consecutive pixels or run length (i.e., a length of thatportion of the current pixel line) for the vertical mode to thetransformation module 314 for subsequent operations or the output module330 for sending. Additionally, for the horizontal mode, the codingmodule 102 may record and transmit a corresponding index value (or pixelvalue) and the number of consecutive pixels (i.e., a length of thatportion of the current pixel line under the horizontal mode) involved inthe horizontal mode to the transformation module 314 for subsequentoperations or the output module 330 for sending.

In some embodiments, upon completing the predictive coding of thecurrent input block (e.g., based on the screen coding scheme or thevideo coding scheme such the intra-picture or inter-picture predictionalgorithm), the transformation module 314 may compute and transform aresidual between the current input block and a prediction result of thecurrent input block as determined based on the predictive coding toobtain a set of transform coefficients using, for example, a lineartransform such as Discrete Cosine Transform (DCT), etc. The quantizationmodule 316 may quantize the transform coefficients which are thenentropy coded by the entropy coding module 324. The output module 330may then send information of color index values representing the majorcolors and the escape pixels, the code obtained in the predictive codingand the transform coefficients which have been entropy coded to theapplication 214 or another client device for decoding or display in theother client device.

In some embodiments, the inverse transformation module 110 may constructthe quantized transform coefficients to obtain an approximation of theresidual, for example, using inverse DCT. The adaptive screen and videocoding system 202 may add the approximated residual to the predictionresult to obtain a summation result. The adaptive screen and videocoding system 202 may smooth out artifacts induced by block-wiseprocessing and quantization using the inverse quantization module 112,the deblocking filter 114 and the sample adaptive offset (SAO) filter116 to obtain a picture representation of the input data (i.e., theplurality of input blocks). The adaptive screen and video coding system202 may store the picture representation(s) of the input data in thedecoded picture buffer 118 to be used for prediction of subsequent inputdata. Additionally or alternatively, the adaptive screen and videocoding system 202 may employ these modules, i.e., the inversetransformation module 110, the inverse quantization module 112, thedeblocking filter 114, the sample adaptive offset (SAO) filter 116 andthe decoded picture buffer 118 for decoding operations.

Example Syntax Element Changes

As described in the foregoing embodiments, the adaptive screen and videocoding system 202 employs a new coding scheme which may lead to syntaxchanges as well as decoding process in an existing video coding schemeor standard. Using HEVC coding scheme as an example, Table 1 showssyntax elements that may be added to PPS (sequence parameter set) andcoding unit (CU), with italic and bold characters representing new oradded syntax elements. In this example, a new flag representing whetherthe screen coding is enabled or disabled, one or more criteria fordetermining whether the screen coding is used for a coding unit (or aninput block), and program codes for determining the one or more majorcolors and escape pixels, etc., are shown in Table 1.

TABLE 1 Example Syntax Element Changes for HEVC Descriptorseq_parameter_set_rbsp( ) { . . .  transform_skip_context_enabled_flagu(1)  intra_block_copy_enabled_flag u(1) intra_screen_coding_enabled_flag u(1) . . . coding_unit(x0, y0,log2CbSize) { . . .  else {   if(intra_block_copy_enabled_flag)   intra_bc_flag[x0][y0] ae(v)   if(!intra_bc_flag[x0][y0]) {   if(screen_coding_enabled_flag)      intra_scm_flag[x0][y0] ae(v)   if(intra_scm_flag[x0][y0]) {   color_index_coding(x0, y0, nCbS, nCbS)  } else {     if(slice_type != I)       pred_mode_flag ae(v)    if(CuPredMode[x0][y0] != MODE_INTRA | |       log2CbSize ==MinCbLog2SizeY)       part_mode ae(v)   }   if(CuPredMode[x0][y0] ==MODE_INTRA) {    if(PartMode == PART_2Nx2N && pcm_enabled_flag &&!intra_bc_flag      && !intra_scm_flag &&      log2CbSize >=Log2MinlpcmCbSizeY &&      log2CbSize >= Log2MaxlpcmCbSizeY)     pcm_flag[x0][y0] ae(v)   . . .   if(!pcm_flag[x0][y0] &&!intra_scm_flag[x0][y0]) {    if(CuPredMode[x0][y0] != MODE_INTRA &&     !(PartMode == PART_2Nx2N && merge_flag[x0][y0]) | |     CuPredMode[x0][y0] == MODE_INTRA && intra_bc_flag[x0][y0])     rqt_root_cbf ae(v)   . . .    }   }  } } color_index_coding(x0, y0,CbWidth, CbHeight) {  for(cldx = 0; cldx < (ChromaArrayType == 0)?1:3;cldx++) {     major_color_num[x0][y0][cldx] ae(v)     for(n = 0; n <major_color_num[cldx]; n++) {      major_color_pred_flag[x0][y0][cldx][n] ae(v)      if(!major_color_pred[cldx])    major_color_pred_dir[x0][y0][cldx][n] ae(v)     }   }  for(cldx = 0;cldx < (ChromaArrayType == 3)?1:3; n++) {   for( i = 0; i < CbHeight;i++) {    line_mode[x0][y0][cldx][i] ae(v)   if(line_mode[x0][y0][cldx][i] == HOR_MODE) {    line_index_flag[x0][y0][cldx][i] ae(v)    if(!line_index_flag[x0][y0][cldx][i]){      for(cSubldx = 0; cSubldx< (ChromaArrayType == 3)?3:1; cSubldx++) {      color_index[x0][y0][cldx + cSubldx][i][0] ae(v)      }     }    }else if{line_mode[x0][y0][cldx][i] = = VER_MODE) {     for(j = 0; j <CWidth; j++) {      (for(cSubldx = 0; cSubldx < (ChromaArrayType = =3)?3:1; cSubldx++) {       color_index[x0][y0][cldx + cSubldx][i][j] =      color_index[x0][y0][cldx + cSubldx][i−1][j]      }     }    } else{     for(j = 0; j < CbWidth; j++) {    index_pred_symbol[x0][y0][cldx][i][j] ae(v)    }    for(j = 0; j <CbWidth; j++) {     if(Indx_pred_symbol[x0][y0][cldx][i][j] = = NOPREDICTION) {      for(cSubldx = 0; cSubldx < (ChromaArrayType = =3)?3:1; cSubldx++) {       color_index[x0][y0][cldx][I][j] ae(v)      }    }    }   }   for(j = 0; j < CbWidth; j++) {    for(cSubldx = 0;cSubldx < (ChromaArrayType = = 3)?3:1; cSubldx++) {    if(color_index[x0][y0][cldx + cSubldx][i][0]      ==major_color_num[x0][y0][cldx + cSubldx]) {      for(j = 0; j < CbWidth;j++)       quant_escape_value[cldx + cSubldx][i][j] ae(v)      }     }   }   }  } }

Examples Methods

FIG. 6 is a flow chart depicting an example method 600 of adaptivescreen and video coding. FIG. 7 is a flow chart depicting an examplemethod 700 of predictive coding. The methods of FIGS. 6 and 7 may, butneed not, be implemented in the framework of FIG. 1 and the environmentof FIG. 2, using the device of FIG. 3, and/or employing methods of FIGS.4 and 5. For ease of explanation, methods 600 and 700 are described withreference to FIGS. 1-5. However, the methods 600 and 700 mayalternatively be implemented in other environments and/or using othersystems.

Methods 600 and 700 are described in the general context ofcomputer-executable instructions. Generally, computer-executableinstructions can include routines, programs, objects, components, datastructures, procedures, modules, functions, and the like that performparticular functions or implement particular abstract data types. Themethods can also be practiced in a distributed computing environmentwhere functions are performed by remote processing devices that arelinked through a communication network. In a distributed computingenvironment, computer-executable instructions may be located in localand/or remote computer storage media, including memory storage devices.

The example methods are illustrated as collections of blocks in alogical flow graph representing a sequence of operations that can beimplemented in hardware, software, firmware, or a combination thereof.The order in which the method is described is not intended to beconstrued as a limitation, and any number of the described method blockscan be combined in any order to implement the method, or alternatemethods. Additionally, individual blocks may be omitted from the methodwithout departing from the spirit and scope of the subject matterdescribed herein. In the context of software, the blocks representcomputer instructions that, when executed by one or more processors,perform the recited operations. In the context of hardware, some or allof the blocks may represent application specific integrated circuits(ASICs) or other physical components that perform the recitedoperations.

Referring to FIG. 6, at block 602, the input module 310 of the adaptivescreen and video coding system 202 receives input data, e.g., an imageof a screen or a video.

At block 604, the input module 310 partitions or divides the input datainto a plurality of input blocks (e.g., a plurality of image blocks).

At block 606, the coding module 102 of the adaptive screen and videocoding system 202 determines which coding scheme is to be used for aninput block of the plurality of input blocks. In one embodiment, thecoding module 102 may determine a coding scheme to be used for the inputblock based on a type of content included in the input block. In oneembodiment, the coding module 102 may determine that a video codingscheme such as an intra-picture prediction coding scheme or aninter-picture prediction coding scheme is to be used for the inputblock. In other embodiments, the coding module 102 may determine that ascreen coding scheme is to be used for the input block.

At block 608, the coding module 102 converts the input block into a codebased on the video coding scheme.

At block 610, the coding module 102 computes a distribution of pixelvalues associated with the input block.

At block 612, the coding module 102 selects one or more pixel valuesfrom the distribution of pixel values as one or more major colors.Additionally, the coding module 102 may treat pixel values that do notbelong to the one or more major colors as escape pixels.

At block 614, the coding module 102 assigns one or more first indexvalues to the one or more major colors. Additionally, the coding module102 may further assign one or more second index values to one or moreescape pixels included in the input block.

At block 616, the coding module 102 classifies each pixel of the firstimage block into a major color of the one or more major colors or anescape pixel of the one or more escape pixels based on a pixel value ofeach pixel.

At block 618, the coding module 102 converts the pixel value of eachpixel into an index value based on which major color or escape pixelthat the pixel value is classified.

At block 620, the coding module 102 exploits or determines correlationsamong the index value of each pixel based on a line-based predictionalgorithm. In one embodiment, the line-based prediction algorithmincludes at least a horizontal mode, a vertical mode and a normal mode.

At block 622, the coding module 102 encodes the input block into a codeaccording to the line-based prediction algorithm.

At block 624, the output module 312 of the adaptive screen and videocoding system 202 sends or streams coding information to a device fordecoding and/or display. For example, the output module 312 may send orstream information of the code the one or more first index values,and/or the one or more second index values to a device for decodingand/or display.

At block 626, the input module 310 determines whether another inputblock or input data exists for processing. If another input block orinput data exists, the coding module 102 processes this other inputblock or input data as described in the above operations.

Referring to FIG. 7, at block 702, the coding module 312 of the adaptivescreen and video coding system 202 receives an index value mapassociated with a plurality of pixel lines of an input block. In oneembodiment, the index value map includes a plurality of joint indexvalues for the plurality of pixel lines, where a joint index value of apixel corresponds to a concatenation of respective index values of colorcomponents of the pixel.

At block 704, the coding module 312 predicts a current pixel line of theinput block based on a previous pixel line or a previous pixel of theinput block.

At block 706, the coding module 312 selects a prediction mode from aplurality of prediction modes based on the prediction. In oneembodiment, the plurality of prediction modes includes a horizontalmode, a vertical mode and a normal mode.

At block 708, the coding module 312 encodes the current pixel line basedon the selected prediction mode.

At block 710, the output module 312 sends or streams a result of theencoding to a device or application (e.g., the client device 204 or theapplication 214) for decoding and/or display.

Any of the acts of any of the methods described herein may beimplemented at least partially by a processor or other electronic devicebased on instructions stored on one or more computer-readable media. Byway of example and not limitation, any of the acts of any of the methodsdescribed herein may be implemented under control of one or moreprocessors configured with executable instructions that may be stored onone or more computer-readable media such as one or more computer storagemedia. Furthermore, the components and operations of the variousembodiments as described in the foregoing may be combined, rearranged,substituted and/or omitted without departing from the presentdisclosure.

CONCLUSION

Although embodiments have been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the claims are not necessarily limited to the specific features oracts described. Rather, the specific features and acts are disclosed asexemplary forms of implementing the claimed subject matter.

1.-20. (canceled)
 21. A computer system, comprising one or moreprocessing units and memory, that implements a video encoder configuredto perform operations comprising: encoding an image to produce encodeddata, wherein the image is partitioned into coding units (“CUs”), andwherein the encoding the image includes, for a given CU of the CUs:determining N major colors for pixel values of the given CU; assigning acolor index value to each of the N major colors and an escape value;converting the pixel values of the given CU to color index values;encoding the color index values for the pixel values of the given CUusing one or more of multiple modes, wherein, according to a predictionmode of the multiple modes, for a given pixel value among the pixelvalues of the given CU, the color index value for a neighboring pixelvalue is used as the color index value for the given pixel value; andfor any of the pixel values of the given CU that is represented with theescape value, encoding the pixel value; and outputting the encoded data.22. The computer system of claim 21, wherein the operations furthercomprise: encoding information that indicates the N major colors,wherein the encoded data that is output includes the encoded informationthat indicates the N major colors.
 23. The computer system of claim 21,wherein, for a given one of the color index values, a quantization zoneincludes a range of pixel values around one of the N major colors, andwherein the converting includes representing any of the pixel values ofthe given CU in the range of pixel values with the given color indexvalue.
 24. The computer system of claim 21, wherein each of the N majorcolors has three color components according to an RGB color model. 25.The computer system of claim 21, wherein the encoding the color indexvalues for the pixel values of the given CU uses, for at least some ofthe pixel values of the given CU, one bit per pixel value to indicate aselection between the prediction mode and a normal mode.
 26. Thecomputer system of claim 25, wherein the encoding the color index valuesfor the pixel values of the given CU further uses a run length toindicate a count of pixel values encoded using the prediction mode or acount of pixel values encoded using the normal mode.
 27. The computersystem of claim 21, wherein the operations further comprise: determiningthat a screen coding scheme is enabled; outputting, as part of asequence parameter set, a flag that indicates the screen coding schemeis enabled; determining to encode the given CU using the screen codingscheme; and outputting a flag that indicates the screen coding scheme isused to encode the given CU.
 28. The computer system of claim 21,wherein, for any of the pixel values of the given CU that is representedwith the escape value, the encoding the pixel value includes quantizingthe pixel value and entropy coding the quantized pixel value.
 29. Thecomputer system of claim 21, wherein, according to the prediction mode,the neighboring pixel value is immediately above the given pixel value.30. The computer system of claim 21, wherein the determining N majorcolors for the pixel values of the given CU includes: determining adistribution of the pixel values of the given CU; and based on thedistribution, determining N most common colors among the pixel values ofthe given CU, for use as the N major colors for the pixel values of thegiven CU.
 31. The computer system of claim 21, wherein the determining Nmajor colors for the pixel values of the given CU includes clusteringthe pixel values of the given CU to find the N major colors, and whereinthe clustering uses k-means clustering.
 32. In a computer system thatincludes a video encoder, a method comprising: encoding an image toproduce encoded data, wherein the image is partitioned into coding units(“CUs”), and wherein the encoding the image includes, for a given CU ofthe CUs: determining N major colors for pixel values of the given CU;assigning a color index value to each of the N major colors and anescape value; converting the pixel values of the given CU to color indexvalues; encoding the color index values for the pixel values of thegiven CU using one or more of multiple modes, wherein, according to aprediction mode of the multiple modes, for a given pixel value among thepixel values of the given CU, the color index value for a neighboringpixel value is used as the color index value for the given pixel value;and for any of the pixel values of the given CU that is represented withthe escape value, encoding the pixel value; and outputting the encodeddata.
 33. The method of claim 32, wherein the encoding the color indexvalues for the pixel values of the given CU uses, for at least some ofthe pixel values of the given CU, one bit per pixel value to indicate aselection between the prediction mode and a normal mode.
 34. The methodof claim 33, wherein the encoding the color index values for the pixelvalues of the given CU further uses a run length to indicate a count ofpixel values encoded using the prediction mode or a count of pixelvalues encoded using the normal mode.
 35. The method of claim 32,further comprising: determining that a screen coding scheme is enabled;outputting, as part of a sequence parameter set, a flag that indicatesthe screen coding scheme is enabled; determining to encode the given CUusing the screen coding scheme; and outputting a flag that indicates thescreen coding scheme is used to encode the given CU.
 36. The method ofclaim 32, wherein, according to the prediction mode, the neighboringpixel value is immediately above the given pixel value.
 37. One or morecomputer-readable media having stored therein computer-executableinstructions for causing one or more processing units, when programmedthereby, to perform operations comprising: encoding an image to produceencoded data, wherein the image is partitioned into coding units(“CUs”), and wherein the encoding the image includes, for a given CU ofthe CUs: determining N major colors for pixel values of the given CU;assigning a color index value to each of the N major colors and anescape value; converting the pixel values of the given CU to color indexvalues; encoding the color index values for the pixel values of thegiven CU using one or more of multiple modes, wherein, according to aprediction mode of the multiple modes, for a given pixel value among thepixel values of the given CU, the color index value for a neighboringpixel value is used as the color index value for the given pixel value;and for any of the pixel values of the given CU that is represented withthe escape value, encoding the pixel value; and outputting the encodeddata
 38. The one or more computer-readable media of claim 37, whereinthe encoding the color index values for the pixel values of the given CUuses, for at least some of the pixel values of the given CU, one bit perpixel value to indicate a selection between the prediction mode and anormal mode.
 39. The one or more computer-readable media of claim 38,wherein the encoding the color index values for the pixel values of thegiven CU further uses a run length to indicate a count of pixel valuesencoded using the prediction mode or a count of pixel values encodedusing the normal mode.
 40. The one or more computer-readable media ofclaim 37, wherein, according to the prediction mode, the neighboringpixel value is immediately above the given pixel value.